Investigate CI failures by Jakob-Naucke · Pull Request #313 · trusted-execution-clusters/operator

Jakob-Naucke · 2026-07-15T15:20:37Z

GHA & openshift/release#79393 saw many random failures, most related to finalizer. Most recent run on the latter showed a resource apparently taking 4 minutes to patch. (Also) testing here for a faster feedback loop.

Summary by Sourcery

Integrate the trustee operator more tightly with an external KBS, switching from in-pod secret mounting to API-based secret and policy synchronization, updating trustee configuration and image versions, and enhancing logging and tests to diagnose and stabilize CI behavior.

New Features:

Introduce JWT-based trustee admin authentication using Ed25519 keys stored in a Kubernetes Secret and wired into the KBS configuration.
Add logic to push and delete machine disk-encryption secrets and attestation keys to the KBS via its API, including a controller to resynchronize them when the trustee deployment becomes available.
Create a dedicated ConfigMap for reference values, plus sync routines to propagate them from Kubernetes to the KBS and to handle recomputation and resync on trustee restart.
Add controllers and helpers to automatically synchronize resource and attestation policies with the KBS and to resync all machine LUKS keys when trustee comes up.

Bug Fixes:

Avoid reliance on in-memory projected volumes for secret delivery by switching to KBS-backed storage, reducing chances of timing-related mount issues during rollout or deletion.],
enhancements:[
Enhance approved image reconciliation and machine/secret finalizers with additional logging and debug information around finalizer execution and cleanup decisions to aid CI failure investigation.
Refine the reference value controller to watch related Jobs, log more details during reconcile, and simplify adoption logic by removing unused image adoption helpers.
Update trustee integration tests to validate end-to-end synchronization of machine secrets and attestation keys with KBS across restarts, and add structured timestamped logging utilities for more traceable test output.
Adjust trustee deployment labels, volumes, and config paths to align with the new KBS storage model and TLS behavior, including support for insecure HTTP mode when no certificate is present.

Build:

Upgrade the default trustee image/tag to v0.20.0 and align Makefile, operator constants, and KBS configuration with the newer release.
Switch the kube-rs dependency to a debug branch to gain better visibility into finalizer behavior during reconciliation.
Enable the env_logger humantime feature to improve log readability in local and CI runs.

CI:

Change the integration test GitHub Actions workflow to run a targeted attestation test with explicit image overrides and to always gather and upload must-gather diagnostics after the job.

Signed-off-by: Roy Kaufman <rkaufman@redhat.com>

The race scenario: - Assume the operator is installed and TEC object exists - Apply an approved-image yaml file (create a new CR instance) - Delete the TEC object (immediately) - The new approved-image is not deleted with the TEC object This happens because kube-rs finalizer() does not call the Apply handler on the first reconcile — it only adds the finalizer and returns. This means adopt_approved_image inside image_add_reconcile never runs on the first reconcile. If the TEC is deleted before the second reconcile, the owner reference is never set and GC (garbage collector) cannot cascade-delete the ApprovedImage. Move adoption before the finalizer() call in image_reconcile so it runs on the very first reconcile. For more information see https://docs.rs/kube/latest/kube/runtime/finalizer/fn.finalizer.html in sections Guarantees, Assumptions (Third point) and Expected Flow(from 1 to 6). Signed-off-by: Roy Kaufman <rkaufman@redhat.com>

All trustee resources, including the resource policy, attestation policy, RV, LUKS key, and AK, are now synchronized using the KBS API. The main motivation for this is replacing the patch mechanism that causes a restart of the trustee deployment. Also, this abstraction over the trustee backend simplifies the process of upgrading to future versions of trustee. Config: - Update kbs-config.toml to v0.20.0 format - Store reference values in dedicated ConfigMap (trustee-rv-data) instead of trustee-data API-driven resource management: - Add KBS API client functions: send_secret, delete_secret, register_ak, sync_resource_policy, sync_attestation_policy, sync_reference_values - Replace volume-mount approach for secrets with KBS API calls - Add trustee deployment reconciler that syncs policies, reference values, and secrets when deployment becomes available Dependencies: - Add kbs-client and jsonwebtoken crates - Bump trustee image tag to v0.20.0 Signed-off-by: Roy Kaufman <rkaufman@redhat.com>

test_luks_key_sync - verify the initial LUKS key upload, re-sync after trustee restart, and LUKS key deletion on machine removal. test_attestation_key_sync - verify that attestation keys are registered with the KBS and re-registered after trustee restarts. Signed-off-by: Roy Kaufman <rkaufman@redhat.com>

Tested functions: - sync_all_machine_luks_key - update_attestation_keys Signed-off-by: Roy Kaufman <rkaufman@redhat.com>

required on OpenShift SCC Signed-off-by: Jakob Naucke <jnaucke@redhat.com>

openshift-ci · 2026-07-15T15:20:42Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: Jakob-Naucke

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

sourcery-ai · 2026-07-15T15:21:20Z

Reviewer's Guide

Refactors the trustee operator to integrate directly with the KBS API (using JWT auth and kbs-client), replaces in‑pod volume-based secret and policy management with explicit sync controllers and config maps/secrets, updates Trustee/KBS configuration and versions, and extends tests and CI to validate LUKS key and attestation key resync behavior and aid debugging intermittent CI failures.

Sequence diagram for trustee deployment sync controller interactions with KBS

sequenceDiagram
    participant Controller as TrusteeSyncController
    participant Kube as KubernetesAPI
    participant KBS

    Controller->>Kube: watch Deployment label app=TRUSTEE_APP_LABEL
    Kube-->>Controller: Deployment with status Available=True
    Controller->>Controller: trustee_deployment_reconcile()

    alt Deployment Available
        Controller->>KBS: sync_resource_policy() / kbs_client::set_resource_policy
        Controller->>KBS: sync_attestation_policy() / kbs_client::set_attestation_policy

        Controller->>Kube: sync_reference_values_from_configmap()
        Controller->>KBS: sync_reference_values() / kbs_client::set_sample_rv

        Controller->>Kube: sync_all_machine_luks_key() / list Machine, get Secret
        Controller->>KBS: send_secret() / kbs_client::set_resource

        Controller->>Kube: update_attestation_keys() / list Secret
        Controller->>KBS: register_ak() / kbs_client::set_sample_rv
    else Deployment not Available
        Controller-->Controller: Action::await_change()
    end

File-Level Changes

Change	Details	Files
Refactor trustee reference values handling to use a dedicated ConfigMap and sync them directly to KBS.	Introduce TRUSTEE_RV_MAP ConfigMap and move reference-values.json storage there instead of trustee-data Make ReferenceValue serializable and deserializable and add helpers to recompute and sync reference values to KBS via kbs_client Add sync_reference_values_from_configmap used by trustee deployment reconcile loop Update image/job reconcile tests to expect TRUSTEE_RV_MAP and extra API calls for KBS sync	`operator/src/trustee.rs` `operator/src/reference_values.rs` `operator/src/kbs-config.toml` `operator/src/main.rs` `operator/src/test_utils.rs` `tests/trusted_execution_cluster.rs`
Introduce JWT-based admin auth for KBS and wire up kbs-client to manage secrets, resource policy, attestation policy, and attestation keys remotely instead of via volumes.	Add Ed25519 keypair generation and TRUSTEE_AUTH_SECRET Secret to hold public/private keys; JWTs issued with jsonwebtoken/jsonwebtoken-openssl Change KBS config to use AuthenticatedAuthorization and LocalJson storage backend under TRUSTEE_STORAGE_DIR Replace mount_secret/unmount_secret and projected volume AK wiring with send_secret/delete_secret/register_ak using kbs_client Add sync_resource_policy and sync_attestation_policy sending rego files over KBS admin API Update attestation_key_register flows to call update_attestation_keys(client) which pushes AKs to KBS rather than patching the trustee Deployment	`operator/src/trustee.rs` `operator/src/kbs-config.toml` `operator/src/main.rs` `operator/src/register_server.rs` `operator/src/attestation_key_register.rs` `operator/Cargo.toml` `Cargo.toml` `operator/src/tpm.rego` `operator/src/test_utils.rs`
Introduce a trustee sync controller to react to trustee Deployment availability and resynchronize state with KBS after restarts.	Add launch_trustee_sync_controller that watches Deployments labeled with TRUSTEE_APP_LABEL and runs trustee_deployment_reconcile In trustee_deployment_reconcile, on Available=True, sequentially sync resource policy, attestation policy, reference values from ConfigMap, all machine LUKS keys, and attestation keys to KBS; requeue on failures Label the trustee Deployment with TRUSTEE_APP_LABEL in generate_kbs_deployment and adjust selector accordingly	`operator/src/trustee.rs` `operator/src/main.rs`
Change machine keygen flow from mounting secrets into the trustee pod to pushing/removing secrets in KBS, with clearer finalizer logging and deletion behavior.	Keygen finalizer now calls generate_secret then send_secret on apply, and delete_secret on cleanup instead of mounting/unmounting volumes into the trustee Deployment Add logging to keygen_reconcile before and after finalizer() for debugging stuck finalizers and cleanups Ensure machine cleanup skips KBS deletion if TrustedExecutionCluster is terminating or missing	`operator/src/register_server.rs` `operator/src/trustee.rs`
Enhance attestation key handling to register AKs in KBS and ensure they are resynced after trustee restarts, and update the OPA policy to check trusted AKs.	update_attestation_keys now lists Secret objects owned by AttestationKey, extracts public_key values, and calls register_ak (kbs_client) instead of mutating the trustee Deployment tpm.rego now requires input.tpm.ak_public to be in trusted_aks reference value Add unit tests for update_attestation_keys success/empty/deleting/error paths and for Ed25519 key generation; extend helper functions to build dummy machines, AK secrets, and trustee auth secrets	`operator/src/trustee.rs` `operator/src/attestation_key_register.rs` `operator/src/tpm.rego` `operator/src/test_utils.rs`
Add integration tests for LUKS key sync and attestation key sync across trustee restarts, and make logging more structured and timestamped.	New tests in trusted_execution_cluster.rs exercise LUKS key KBS sync on machine creation, trustee restart resync, and deletion, and similar flows for attestation key registration and resync Extend TestContext with wait_for_deployment_ready helper and prepend timestamps to test_info/test_warn output Adjust existing integration tests to use TRUSTEE_RV_MAP instead of trustee-data for reference values and ensure teardown waits for RV ConfigMap deletion	`tests/trusted_execution_cluster.rs` `test_utils/src/lib.rs`
Update versions and CI pipeline to use newer Trustee image/config and to focus integration CI on the attestation test with better debug artifacts.	Bump TRUSTEE_VERSION and default TRUSTEE_IMAGE to v0.20.0 and align KBS configuration with new API (jwt admin, kvstorage, LocalJson storage) Switch kube dependency to a debug branch finalizer-dbg to gain additional finalizer diagnostics Modify integration-tests GitHub workflow to generate CRDs, run a focused attestation test with verbose logging, and always collect and upload must-gather artifacts	`operator/src/main.rs` `Makefile` `operator/src/kbs-config.toml` `.github/workflows/integration-tests.yml` `Cargo.toml` `Cargo.lock`

Tips and commands

Interacting with Sourcery

Trigger a new review: Comment @sourcery-ai review on the pull request.
Continue discussions: Reply directly to Sourcery's review comments.
Generate a GitHub issue from a review comment: Ask Sourcery to create an
issue from a review comment by replying to it. You can also reply to a
review comment with @sourcery-ai issue to create an issue from it.
Generate a pull request title: Write @sourcery-ai anywhere in the pull
request title to generate a title at any time. You can also comment
@sourcery-ai title on the pull request to (re-)generate the title at any time.
Generate a pull request summary: Write @sourcery-ai summary anywhere in
the pull request body to generate a PR summary at any time exactly where you
want it. You can also comment @sourcery-ai summary on the pull request to
(re-)generate the summary at any time.
Generate reviewer's guide: Comment @sourcery-ai guide on the pull
request to (re-)generate the reviewer's guide at any time.
Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
pull request to resolve all Sourcery comments. Useful if you've already
addressed all the comments and don't want to see them anymore.
Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
request to dismiss all existing Sourcery reviews. Especially useful if you
want to start fresh with a new review - don't forget to comment
@sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

Enable or disable review features such as the Sourcery-generated pull request
summary, the reviewer's guide, and others.
Change the review language.
Add, remove or edit custom review instructions.
Adjust other review settings.

Getting Help

Contact our support team for questions or feedback.
Visit our documentation for detailed guides and information.
Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.

Signed-off-by: Jakob Naucke <jnaucke@redhat.com>

for crates and testing Fixes: trusted-execution-clusters#227 Signed-off-by: Jakob Naucke <jnaucke@redhat.com>

so that status is updated in a timely manner Signed-off-by: Jakob Naucke <jnaucke@redhat.com>

Signed-off-by: Jakob Naucke <jnaucke@redhat.com>

Signed-off-by: Jakob Naucke <jnaucke@redhat.com> Assisted-by: AI

GHA has been seen to run out of space. Add a flag to delete local image after push. Signed-off-by: Jakob Naucke <jnaucke@redhat.com> Assisted-by: AI

iroykaufman and others added 6 commits July 15, 2026 16:40

trustee: Add key pair as a base to use kbs API

cceffec

Signed-off-by: Roy Kaufman <rkaufman@redhat.com>

trustee: Add unit tests

e035c3b

Tested functions: - sync_all_machine_luks_key - update_attestation_keys Signed-off-by: Roy Kaufman <rkaufman@redhat.com>

trustee: Add empty directory volume for storage

f3686b3

required on OpenShift SCC Signed-off-by: Jakob Naucke <jnaucke@redhat.com>

openshift-ci Bot added the do-not-merge/work-in-progress label Jul 15, 2026

Jakob-Naucke force-pushed the investigate-patch branch 3 times, most recently from 1353d67 to 4de09c8 Compare July 16, 2026 09:00

Jakob-Naucke added 6 commits July 16, 2026 12:38

trustee: try pad-token

d3aee0a

Signed-off-by: Jakob Naucke <jnaucke@redhat.com>

Add timestamps to logging

f0ee3a0

for crates and testing Fixes: trusted-execution-clusters#227 Signed-off-by: Jakob Naucke <jnaucke@redhat.com>

rvs: Reconcile images on jobs too

ccbc432

so that status is updated in a timely manner Signed-off-by: Jakob Naucke <jnaucke@redhat.com>

Finalizer calling logging

c56e1c5

Signed-off-by: Jakob Naucke <jnaucke@redhat.com>

gha: run & upload must-gather

f1934b6

Signed-off-by: Jakob Naucke <jnaucke@redhat.com> Assisted-by: AI

make: add DELETE_AFTER_PUSH flag

00e5c14

GHA has been seen to run out of space. Add a flag to delete local image after push. Signed-off-by: Jakob Naucke <jnaucke@redhat.com> Assisted-by: AI

Jakob-Naucke force-pushed the investigate-patch branch from 4de09c8 to 00e5c14 Compare July 16, 2026 10:38

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Investigate CI failures#313

Investigate CI failures#313
Jakob-Naucke wants to merge 12 commits into
trusted-execution-clusters:mainfrom
Jakob-Naucke:investigate-patch

Jakob-Naucke commented Jul 15, 2026 •

edited by sourcery-ai Bot

Loading

Uh oh!

openshift-ci Bot commented Jul 15, 2026

Uh oh!

sourcery-ai Bot commented Jul 15, 2026

Interacting with Sourcery

Customizing Your Experience

Getting Help

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

Jakob-Naucke commented Jul 15, 2026 • edited by sourcery-ai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary by Sourcery

Uh oh!

openshift-ci Bot commented Jul 15, 2026

Uh oh!

sourcery-ai Bot commented Jul 15, 2026

Reviewer's Guide

Sequence diagram for trustee deployment sync controller interactions with KBS

File-Level Changes

Interacting with Sourcery

Customizing Your Experience

Getting Help

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Jakob-Naucke commented Jul 15, 2026 •

edited by sourcery-ai Bot

Loading